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Engineers' love of technology often gets 
in the way of their being useful. Consider 
Posl-it Notes OT, better yet, plain paper note- 
pads. These probably seemed like trivial 
ideas, but they turned out to be terribly use- 
ful. Why? Because the marvel thai is the 
bunian brain has a horrible short-tcrai mem- 
ory, which means that dumb-as-dirt memory 
aids can make people substantially smarter. 



The human pan of any large intelligent 
system is by far the most inielligeni pan. 
As long as this remains true, the biggest 
system advancements will come from aids 
that fill big holes in human abilities, rather 
than from artifacts that stretch engineers' 
abilities. 

I nwndon all this because I want you to 
consider a simple, not very technically chal- 
IcQging idea — one that might neverthelcM 
fill a gaping hole in our collective intelli- 
gence, similar to the way notepads fill a 



gaping hole in our individual memories. I 
am talking in general about speculative mar- 
kets, and in particular about decision mar- 
kets. Decision markets might allow us to 
more accitrately estimate the consequences 
of important decisions, by helping us to bet- 
ter share relevant information. 

Consider, for example, a clearly impor- 
tant policy question such as, 

How would crime rates change if more citi- 
zens could legally carry hidden guns? 

Many observers say hidden guns obvi- 
ously increase crime, while many others 
strongly disagree (see John Lou's More 
Guns. Less Crime, Univ. of Chicago 
Press). 

I suspect that the existence of .^uch diver- 
gent opinions reflects the fact Uiai we suf- 
fer from a serious failure to share informa- 
tion. The so-called Information Revolution 
has greatly improved our ability to fmd out 
what others have said. However, it has done 
much less to improve our ability to find out 
what other people know. We can now find a 
blizzard of words on a topic such as the 
interplay between guns and crime, but we 
know that most of those words are written 
by people with axes to grind. The real 
problem is not finding more words, but 
judging who really knows ^ut the topic 
and whether diese experts arc saying what 
they know. We are in many ways bit-full, 
yet information-poor. 

I suggest that speculative markets are a 
neglected way to help us find out what peo- 
ple know. Such markets pool the informa- 
tion that is known to diverse individuals 
into a common resource, and have many 
advantages over standard institutions for 
information aggregation, such as news 
media, peer review, trials, and opinion 
polls. Speculative markets are decentral- 
ized and relatively egalitarian, and can 
offer direct, concise, timely, and precise 
estimates in answer to questions we pose. 
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These estimates are self-consistent across a 
wide range of issues and respond quickly 
10 new information. They also seem to be 
cheap and relatively accurate. 

In particular, for questions such as hid- 
den guns and crime, I suggest we consider 
decision markets, which are speculative 
markets focusing on particular decisions. 

How dedsUw markets wotk 

hnagine thai we created markets where 
people could bet on ftimre ciime rates, con- 
ditional on allowing or not allowing more 
hidden guns. TTiat is, if the market prices 
predicted that murder rates would be 10% 
higher should a certain hidden-gun bill pass, 
then anyone who thought this estimate too 
high could easily identify a particular prof- 
itable trade, [f you made this trade and the 
tnarket estimate then fell to 5%, for exam- 
ple, you could undo this trade for a profit. 

Imagine further that if market prices said 
that crime rates would be 10% higher given 
more hidden guns, most nonexperts would 
accept this estimate as our "best answer" or 
as a neutral "consensus." In particular, a 
state legislature might accept this estimate 
when considering whether or not to pass 
this hidden-gun bill. 

1 call such a set of markets a decision 
market. In this situation, advocates for each 
side of an issue would be forced to influ- 
ence speculators if they wanted to influence 
general opinion. Speculators, in turn, would 
have a clear incentive to be careful and 
honest in contributing what they know and 
in judging what advocates know. This is 
because speculators must '*put their money 
where their mouth is." 

Six steps are required to create a deci- 
sion market to help us belter share informa- 
tion on a topic such as the effect of hidden 
guns on crime. 

First, you must stale your claim clearly. 
For example, you might focus on a particu- 
lar bill B before your state legislature, 
which would allow more citizens to carry 
hidden guns. You might decide to focus on 
your state's murder rate, using some stan- 
dard government statistic M as your oHicial 
measure of it. You should choose a lowest 
and highest relevant murder rate, and scale 
M so that M = 0 at the lowest rate and M = 
1 at the highest rate. (You might choose, 
for example, the lowest rate to be zero mur- 
ders and the highest rate to be the popula- 
tion size, which is the highest conceivable 
murder rate.) 



Next, you must choose some paniculra- 
trusted third party who will finally declare 
a murder rate M within [0. 1 ] and determine 
whether bill B pas.«;ed. (This party might be 
a jury randomly drawn from some pool.) 
You must also either pick a date by which 
the.se judges are to decide, lie the judging 
to some other event like the release of mur- 
der statistics, or grant the judges discretion 
to choose this date for themselves. 

Third, you must choose what asset A you 
will bet If the bet is going to last any sub- 
stantial time, asset A ought to give a rea- 
sonable rate of return, to induce speculators 
to invest in it. You might pick a safe gov- 
ernment bond, or you might pick a broad- 
index stock mutual fund. 

Also, given M, B, and this asset A. you 
authorize some financial instiiuiiun to 
make exchanges between units of A and the 
following set of four assets: 

1. M units of A if B passes. 

2. M units of A if B does not pass. 

3. 1 - M units of A if B passes, 

4. I - M units of A if B does not pass. 

Note that because M + (1 - M) = 1 , and be- 
cause B either will or won't pass, this fioan- 
cial institution takes no risk from these 
exchanges; each set of four assets will be 
worth exactly one unit of A in the end. 

Fifth, you create markets in which peo- 
ple can trade various combinations of these 
assets with each other. In particular, if peo- 
ple trade asset 1 for the bundle of assets 1 
and 3, the market price (asset ratio in 
trades) is an estimate of the murder rate 
conditional on the bill passing. Similarly, 
the price in trades of asset 2 for the bundle 
of 2 and 4 is an estimate of the murder rate 



conditional on the bill not passing. More- 
over, comparing these two estimates tells 
you whether, and by how much, specula- 
tors expect this bill to increase or decrease 
the murder rate. 

Finally, you have to decide how much to 
subsidize this market. If interest in your 
topic is strong enough, simply creating 
these markets might induce people to trade 
in them. Failing that, sufficient interest 
might be induced if someone committed to 
make a policy choice based on the market 
estimate. A slate legislature, for example, 
might commh to pass the bill or not de- 
pending on the market estimate of their 
effect on the murder rate. 

You can also safely and directly subsi- 
dize a market to induce more participation. 
Doing this in effect creates an information 
prize, offered lo those who first make the 
market price better reflect relevant infor- 
mation. (In econo-speak, one way to do 
this is to create a market maker whose bid 
and ask prices are monotonic functions of 
its assets held.) 

In addition to estimating the effect of 
hidden guns on crime, decision markets 
might give us estimates on 

• Murder rates — with or without capital 
puni.ihmeni? 

• Average mortality rates — with or with- 
out national health insurance? 

• Health-care spending — with expanded 
or curtailed use of health mai menace 
organizations? 

• Employment change — raise minimum 
wage or rescind NAFTA? 

• Global sea-level and temperature 

.; changes — impose or not impose a car- 
I bon dioxide tax? 
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• Military casualties — with a 
Republican or a I>emocratic' 
president? 

• Stock prices — with a Republi- 
can- or Democral-controlled 
US Congress? 

• World per-capita food con- 
sumption — raise or lower aver- 
age tariffs? 

• Student test scores — with or 
without school choice or 
voucher reform? 

• Future national economic growth 
— raise or lower interest rates, 
or with or without an education 
subsidy? 

Science fiction writers have pos- 
ited even more ubiquitous betting 
markets (see John Brunner's Shock- 
wave Rider, Del Rcy Books, and Marc Stei- 
gler's Earthweh). In general, decision mar- 
kets can estimate the net effect of any policy 
choice of interest on any outcome of interest, 
as long as there is a decent chance that, after 
the fact, we can reasonable veri^' what out- 
come happened and wb^ policy was chosen. 

How wall do markets work? 

By its nature, a bctting-inarket estimate 
is decentralized, direct, concise, timely, 
precise, self-consistent, and responds 
quickly to new information. It is also egali- 
tarian, if everyone is allowed to participate. 
But how clear, cheap, and accurate arc such 
market estimates? 

On accuracy, decades of research on the 
efficiency of financial markets have found 
litUc price-relevant information that is not 
reflected in market prices. Any inefTicien- 
cies seem to be weak and to go away with 
publicity, because they represent a profit 
opponunity. If you think the current price 
is too low. you expect to profit by buying 
now and selling later, and buying now will 
raise the price, partially correcting the error 
you perceived. 

Speculative markets have done well In 
direct te.sts against standard information- 
aggregation institutions. For example, 
orange juice futures prices have been shown 
to improve on government weather fore- 
casts.' Also, markets where traders can bet 
on election results predict vote totals belter 
than opinion polls. - 

How do markets do so well? After all. 
aren't ihey made of the same fallible 
humans as other institutions? A study of 
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those election markets found that while 
most traders tended to siiffer from cogni- 
uve biases such as'expccting others to 
agree with them, the most active traders 
were not biased this way — and active 
traders set the prices. Speculative markets 
thus seem to induce the real experts to self- 
select and participate more. Lab experi- 
ments also indicate that speculative mar- 
kets tend 10 aggregate information when 
traders are experienced with their roles and 
know the payoffs for other roles.' 

Older economics writings sometimes 
give the impression that speculative mar- 
kets will not function unless you have thou- 
sands of traders ErequenUy trading million.s 
of dollars worth of goods. Recent Web 
markets, however, show clearly that mar- 
kets can be much smaller and slower than 
this. Furthermore, a subsidizixl market can 
funcdon over any time period with only 
one trader. If an information prize is 
offered, and only one person is induced to 
learn enough to only once correct the initial 
market price to something else, the market 
has still ser\'ed a valuable informadon role. 

Widi subsidies, the key question is not 
whether we can create speculative markets, 
or whether such markets can induce people 
to leara and reveal information. The key 
questions are whether the information 
gained is worth the costs paid and whether 
a similar benefit could have come cheaper 
via some other instimtion. 

Whafsriie holdup? 

If speculative markets are so great at in- 
formation aggregation, why don't we al- 



ready use them to form consensus 
on topics such as. the correlation 
between hidden guns and crime? 

Until the last few ceniuries. the 
cost of simply handling trades was 
enough to sharply limit the number 
of speculative markets that could be 
made widely available. Yet today 
ebay. com routinely sells $10 items 
by having a handful of people bid a 
few times each over a period of a 
week. Moreover, play -money Web 
betting games have shown that just 
a handful of people, each making a 
few small trades over several years, 
can create reasonable esdmates on 
a wide variety of questions. (See 
www.hsx.com, myhand.com, and 
' espcciMywww.UUosphere.com.) 
More important, most speculative 
markets are now illegal. The shon history of 
financial market regulation is that every- 
thing was once illegal, until limited exemp- 
tions were granted for specific purposes. 
Betting on cards was a foolish waste of 
money; only fools would invest in a business 
they did not closely monitor; and it was the 
height of folly to let people bet on die death 
of others. So casinos, stocks, and insurance 
were all harmed. * 

Gradually, exceptions were granted for 
what came to be seen as worthy purposes, 
such as teaching people about horses 
(horseracing) or raising state revenue (lot- 
teries). Stocks were allowed for the pur- 
pose of capitalizing firms, and insurance 
was allowed to let individuals hedge risks. 
More recently, commodity fuuires and 
financial derivatives were allowed to let 
firms hedge more risks. All these areas are 
highly regulated, however, in part to pre- 
vent limited exemptions from devolving 
into general gambling. 

Accepted functions of markets now 
include entertainment, capitalization, and 
hedging, but not information aggregation. 
Thus while ii is widely recognized thai 
markets created for other purposes accom- 
plish LA. we're prevented from creating a 
market whase primary function is lA. So 
we cannot create a market whose legal 
price would inform raging policy debates, 
such as ihe interplay between crime rates 
and hidden guns. 

Okay, betting markets are niainly illegal. 
But if economists hove data suggesting that 
that speculative markets do well at lA, why 
aren't lots of economists pushing the idea 
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of better via more maitets? 

AcUially. it's worse than you think; econ- 
omists also have sophisticated theory that 
suggests that lA should not be that hard on 
factual topics like the effect of hidden guns 
on crime. Rational agents should not even 
be able to agree on which one of them thinks 
hidden guns cause more crime.' (I won't 
say more, as the editor wisely advises 
against using more econo-speak.) Eco- 
nomic theory thus really does suggest we 
humans have a gaping hole in our social 
intelligence. 

So why aren't economists pushing lA 
markets? One answer is that economists are 
just spread too thin. Economic theory sug- 
gests many jrolicy improvements over the 
status quo, and there are few economists 
that anyone else will listen lo. These few 
economists thus have to choose their bat- 
tles carefully. 

The relevant theorj' for I A is also recent, 
and most economists don't yet know about 
it. Worse, the I A functions of markeLs 
seem too complex to model in much gen- 
erality. When systems become too com- 
plex to model in detail, engineers usually 
resort to building and testing theory- 
inspired prototypes. However, economic- 
theorist types who understand this area are 
reluctant to move that far away from the- 
ory. (Capitalization and hedging functions 
of markets are easier to model, and econo- 
mists do use theory to design market pro- 
totypes for these functions.) 

ll thus seems to fall to a few economics- 
savvy and engineering-minded folks like 
me to think of using prototypes to explore 
the idea of using more speculative markets 
for lA. 

A proRiIffaig direction: Internal 
forpomte marketi 

On the types of topics to which they 
have been applied so far, lA markets have 
looked promising. There remain, however, 
many legitimate concerns. For example, 
does the existence of speculative markets 
discourage commimication via other chan- 
nels, and is this a net benefit or loss? 

Tests of prototypes might help us answer 
such questions. But how can we test proto- 
types, if LA markets are generally illegal? 
Well, there is one plausible loophole (be- 
sides offshore gambling), which I have 
saved for those of you who arc still reading 
this fan internal corporate markets. Corpo- 
rations have great leeway in what they make 



employee bonuses depend on. and a contin- 
gent bonus is pretty close to a bet. So several 
companies, including Hewlett-Packard and 
Siemens, have begun experimenting with 
real-money internal speculative markets for 
estimating things such as future sales. 

Corpomtions also need to make deci- 
sions, and often have problems inducing 
relevant parties to reveal information about 
the consequences of those decisions. Fur- 
thermore, companies have a good rough- 
and-ready measure of "good for the com- 
pany" — the stock price. Thus, you could 
create decision markets that estimate 
whether any particular decision, such as 
introducing a new product, is better or 
worse than some alternative for the stoct 
price. Alternatively, you might predict the 
sales of some prtxiuct contingent on some 
important product-design decision. 

Just as notepads fill a gaping hole in our 
individual cognitive abilities, speculative 
markets might fill a gaping hole in our col- 
lective ability to share information. Eco- 
nomic theory suggests tiiat lA should not 
be that hard, at least for factual policy 
questions like the effect of hidden guns on 
crime rates. Speculative markets seem to 
work well at such tasks. Let us thus de- 
velop prototypes to explore this potential, 
in the hopes of someday lifting cunent 
legal barriers to widespread use of more 
effective institutions for I A. 

For more infonnation on this topic, see 
http://fujnson.bf.rk£ley.edu/id£afuturesJuml. 
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After decades of research in paitcm 
recognidon and components of intelligent 
systems, the AI conununity has shifted its 
focus from fundamental concepts and math- 
ematical techniques to large-scale data ac- 
quisition and knowledge en^neering. In 
contest after contest in academic and com- 
mercial realms, the best systems for optical 
character, speech and face recognition, and 
so on are the ones trained with the most 
data. Collecting very large, high-quality 
datasets is evidently vital to progress in 
these and several other areas. Such data is 
informal — known by everyone who can 
read, speak, or hear, or has a commonsense 
understanding of the worid. 

Consider, too, the qualified but increas- 
ingly compelling success of the Open 
Source methodology — which promotes 
software reliability and quality by support- 
ing independent peer review and rapid evo- 
' lulion of freely distributed source code. 
Linux. SendMail. Apache, the Mozilla ver- 
sion of the Netscape Web browser, and 
other high-quality software testify to the 
viability of this collaborative approach to 
software engineering. 

Together, these developments suggest a 
new approach lo building components of 
intelligent systems— the Open Mind Initia- 
tive. This initiative relies on dircc types of 
participants: 

• domain experts who contribute libraries 
of algorithms, 

• tool developers who contribute and 
refine the enabling software, and 

• lay c-citizens who contribute data via 
the Internet. 

Users with an interest and expenise in a 
particular domain, such as speech, vision. 
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language, or commonscnse, serve as re- 
viewers or moderators. 

Consider the development of an optical 
character-recognition system through Open 
Mind (see Figure I). A host machine pre- 
sents pixel images of handwritten charac- 
ters, transformed by distortions, warping, 
line thickening, and so forth, on browsers 
of e-ciiizens. perhaps in a game interface. 
E-citizcns classify these images by means 
of a button response; the responses are 
aggregated, screened for significant out- 
liers at the host machine, and used as train- 
ing data to improve the classifier. 

To explore the novel infrastructure re- 
quired by Open Mind, Stanford graduate 
student Chuck Lam and I have developed a 
Java- and Web-based version Animals, 
an elementary interactive children's pro- 
gran) for classifying animals, dating from 
the late 1970s. The child (e-citizcn) thinks 
of an animal. Using the child's responses to 
a series of questions (for example, two- 
leofled or four-legged?), the program 
tries to detenminc this animal's identity. If 
the program guesses wrong, the child must 
enter a question that distinguishes her ani- 
■ nna) from the program'.s guess. After a 
number of children have played this guess- 
ing game. Animals has learned a simple 
tree -based classifier for animals. l^Tiile not 
a deep or particularly useful core program, 
Animals provides an excellent platform for 
solving important problems in Open Mind. 



It provides an interface and protocol design 
for 

• efficiently extracting the maximum 
information from c-cidzens, 

• detecting and eliminating significant 
errors and statistical "outliers." 

• repelling hostile attacks, and 

• automatically listing contributors 
according to the amount of information 
they contributed. 

The Open Mind Initiative differs from 
the Free Software Foundations and the 
Open Source organization in a number of 
ways. First, while Open Source draws its 
support almost entirely from a hacker cul- 
ture (for example, roughly 10^ program- 
mers have contributed to Linux), Open 
Mind relies heavily on an e-citizcn and 
business culture ( 1 0' nonprogrammers on 
the Web). While most of the work in Open 
Source is directly on the final released 
source code, most of the effort in Open 
Miod focuses on the tools, infrastructure, 
and data gathering. An expert arbitrates 
final decisions in Open Source; in Open 
Mind, much information is accepted or 
rejected automatically by the infrastructure 
software. Table 1 summarizes some of 
these differences. 

While domain experts and infrBstructure 
developers are likely to have the same mo- 
tivations as contributors to Open Source, the 



Table 1 . Comparisons between Open Source and Open Mind. 



OpENStKDlCE 



Open Mho 



No e-citizans 

Expert knov/ledge 

Machine learning irrelevant 

Web useful but optional 

Most wortt is (jirectly on ttie end-user 

software 
Hacker culture (<10^} 
Separate functions contributeci 

(device drivers in Linux) 



E-citizens crucial 
Informal knowledge 
Machine learning essential 
Web essential 

Most work is on intrastnjctura, not on the 

end-user software 
E-citizen/business culture (< 10^) 
Single functional goal (for example, recognition 

rate in OCR) 



motivations of e-citizens deserve special 
consideration. E-citizens seek benefit from 
the resulting Open Mind sofbvare, including 
software that would be very difficult to de- 
velop in other ways, such as commonsense 
knowledge. E-citizens would enjoy game 
interface.*; and seek the public reaigniiion of 
their contributions, lliere could be financial 
incentives such as lotteries, discounts, or 
frequent-flier awards provided by corpora- 
tions seeking new customers. 

Others have discussed components of 
Open Mind and its use in different con- 
texts—online interactive data acquisition 
and voting, collaboration, and machine- 
learning and pattern-recognition algo- 
rithms—yet the integration proposed in 
Open Mind seems not to have been dis- 
cussed. A number of existing projects 
would fit under an Open Mind umbrella and 
would profit from the initiative. One exam- 
ple is Newhoo, where nonspecialist e-citi- 
zens propose keyword and index informa- 
tion about Web pages. Their contributions 
are reviewed by volunteer referee and edi- 
tors (currently 10,000) and made available 
to all. We can imagine Open Mind projects 
in numerous pattern-recognition domains or 
knowledge engineering to improve navigat- 
ing news groups, Web sites, or FAQs. Later, 
these systems can be integrated, for in- 
stance, to incorporale natural language or 
commonsense constraints in speech recog- 
nition. OCR, or web searching. 

Physics has had its atom smashers, 
microbiology its Human Genome Project, 
and aeronautics and astninautics; its space 
missions. Now is the time for computer 
science and cognitive science to have 
their big science— one that harvests infor- 
mal knowledge from a large number of 
c-citizcns for building useful software for 
nexi-generation systems. Given the con- 
junction of several forces — the need for 
natural human-machine interfaces and 
improved Web .searching, the existence of 
good learning algorithms and Web infra- 
structure, and the demonstrated success of 
the Open Source methodology — the tittK 
is right for the Op»en Mind Initiative. 
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Abstract: 

We describe the Open Mind Initiative, a framework for large scale collaborative efforts In building 
components of "intelligent" systems that address common sense reasoning, document and language 
understanding, speech and character recognition, and so on. Based on the Open Source methodology, the 
Open Mind Initiative allows domain specialists to contribute algorithms, tool developers to provide 
software infrastructure and tools, and non specialist "e-citizens" to contribute training data and 
information to large databases. An important challenge is to make it easy and rewarding for e-citizens to 
provide such information. The paper illustrates the initiative through several demonstration projects of 
modest scale, including some related to character and document problems, and identifies general 
challenges and opportunities. 
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Abstract: 

The article comprises two sections. In the first, the author suggests that speculative markets are a 
neglected way to help Us find out what people know. Such markets pool the Information that is known to 
diverse individuals into a common resource, and have many advantages over standard institutions for 
information aggregation, such as news media, peer review, trials, and opinion polls. Speculative markets 
are decentralized and relatively egalitarian, and can offer direct, concise, timely, and precise estimates in 
answer to questions we pose. In the second section, the author argues that now Is the time for computer 
science and cognitive science to have their big science-one that harvests informal knowledge from a large 
number of e-citizens for building useful software for next generation systems. Given the conjunction of 
several forces-the need for natural human-machine interfaces and improved Web searching, the existence 
of good learning algorithms and Web infrastructure, and the demonstrated success of the Open Source 
methodology-the time is right for the Open Mind Initiative. 
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